On the Importance of Hubs in Hopfield Complex Neuronal Networks under Attack 
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The organizational principles behind the connectivity of a complex network are known to influence 
its behavior. In this work we investigate, using the Hopfield model, the influence of the network 
architecture on the performance for associative recall while the network is under hub and edge 
attack. We show, by using four different attack strategies, that although the importance of hubs is 
more definite for Barabasi- Albert neuronal networks, the random removal of the same amount of 
edges as in a hub may imply a greater reduction of memory recall. 



00 
(N 

a 



I 

a 

o 

o 



> 

l> 
\o 
l> 
o 
in 
o 

» 

03 



i 

a 

o 

o 



X 



PACS numbers: 



I. INTRODUCTION 



A great deal of the current interest on networks, neu- 
ronal or complex, stems from the fact that the function- 
ality of network systems is highly influenced, and even 
determined, by their respective connectivity (e.g. 0). 
While most neuronal network models were initially fully 
connected, increasing interest has been focused more re- 
cently on systems with diluted (e. g. Il3l. J T 3 . I22I l23j l 
and/or structured (e.g. |i FR 111 Il7l 12011251 W\. |2S[ ) con- 
nectivity. A series of interesting points and questions 
are implied by such investigations, especially regarding 
structured networks of connections, i.e. models departing 
from uniformly random connections. For instance, given 
that metabolic and spatial biological pressures constrain- 
ing the number of dendrites and axons in a neuronal cell, 
it is virtually impossible to have fully-connected biolog- 
ical networks (e.g. 0]). What are the functional impli- 
cations of such unavoidable partial connections? Which 
partially connected structure (e.g. random, small world, 
scale free) allows the best functionalities? Are hubs par- 
ticularly important for scale-free neuronal networks? In 
which sense they affect the recall of patterns in associa- 
tive networks such as the Hopfield model? 

Because they concentrate connections, hubs have been 
found to play a fundamental role in definin g th e connec- 
tivity of scale free complex networks (e.g. pl lllLl24| ). For 
instance, as they connected to many nodes, hubs provide 
bypasses between a large number of pair of nodes, con- 
tributing to the overall reduction of the mean shortest 
path in the network. It is intuitive to expect that hubs 
would also be particularly important for the dynamics of 
systems with scale-free connectivity, such as the Hopfield 
complex neuronal networks considered in 0, [25| . Such 
a question provided the main motivation for the present 
article. In order to try to answer it, we performed node 
and edge attack on random and Hopfield complex net- 
works and monitor the effect over the recall performance. 
Among the reported experiments, we successively remove 



hubs from scale free Hopfield models, and then remove 
the same number of edges by node and edge attack on 
random models. Although hubs are confirmed to have 
special importance for the memory capacity of the net- 
work, it was remarkably found that their removal is not so 
impacting as the random elimination of the same number 
of edges from complex or random networks. One expla- 
nation for such a phenomenon is the fact that a hub is 
most directly and strongly associated to a single bit of 
the patterns to be trained and recovered. 

The article starts by reviewing the Hopfield model and 
complex networks, and follows by describing the simula- 
tion framework and the obtained results. It concludes by 
interpreting and discussing the results and their implica- 
tions for further researches. 



II. METHODOLOGY 

The Hopfield model 

is a model of autoassociative 
memory which has been studied and generalized thor- 
oughly since its first appearance. Today, the value of the 
original model is mostly epistemologic, as practical im- 
plementations are constrained by several issues such as 
limited efficiency for storing several patterns. Neverthe- 
less, being a simple and intuitive model, it provides an 
effective framework for performance studies under vary- 
ing conditions while allowing the quantification of the 
impact of the choice of specific influencing elements. 

In this paper we follow closely the implementation de- 
scribed by [l5| . A set of M binary string patterns are 
represented by vectors as Pi = \pi,P2, ■■■,Pn] T , whose N 
elements are in either of the possible states pj = ±1, 
randomly chosen. This information is stored in a weight 
matrix W, according to the follow expression 
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In order to retrieve a pattern stored initially, say Q n = 
[qi, 52, q n ] T , it is necessary to iterate the following ex- 
pression 
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where sng [.] is a hard limiting function. In this imple- 
mentation, whenever the value of the state of a neuron 
becomes exactly zero, the previous state of that neuron 
is used instead, so that the neurons are always either fir- 
ing (+1) or quiescent (-1). The element Wij of W is the 
weight of the connection of neuron i with neuron j with 
the W matrix representing a fully connected network, 
except for loops (i.e. wu = 0). 

A diluted version of the Hopfield model still retaining 
pattern recognition properties, see p5|. can be obtained 
by multiplying, in elementwise fashion, the weight matrix 
by a sparse weight connection matrix, build upon some 
assumed heuristics. 

We consider in this paper two models of complex 
networks, namely the random and Barabasi-Albert net- 
works, as connection matrices underlying Hopfield mod- 
els. A network with N nodes can be represented by a 
symmetric adjacency matrix C of rank N, with each el- 
ement Cij assuming unit value for connected nodes and 
null values for disconnected ones. We set cu = to pre- 
vent self connections. The random network is obtained 
simply by choosing connections Cy = 1 with probability 
7. The parameter 7 therefore determines the density of 
connections. 

In the Barabasi-Albert model P, Q , we start with a 
small random network A with mo nodes. For each node 
cii the probability pi of a new connection is given as 
k( a i)/ J2i(k( a i))> where fc(aj) is the node degree. Con- 
sider a new node b with m edges. A node cij of the initial 
network A is allowed to connect to one of the m edges 
of the new node b with the probability pi. In other 
words, the degree of node defines its chances of re- 
ceiving new connections, suggesting the paradigm 'the 
rich gets richer'. 

III. SIMULATIONS 

In this section we consider both kinds of network ar- 
chitectures (i.e. random and scale free), characterized by 
their respective connection matrix C, as defined in the 
previous section. The training matrix W of the Hopfield 
model, calculated for a initial set of random patterns P n , 
is diluted by direct multiplication with the connection 
matrix C of the network under analysis. The pattern Pi 
will stand for the reference pattern for these experiments. 
We add 10% of noise to Pi before trying to recover Pi 
through the iterative recall process, see Equation [3 Af- 
ter fifty interactions, an overlap index is calculated, mea- 
suring how much of the recovered pattern P r matches 
the original pattern Pi. This process is repeated for an 
increasing number of patterns Pi, i = 1,2, ...,M. The 
calculation of the overlap index is performed by 

Overlap® = {Pr ^ lK% \ (3) 

where N is the total number of neurons in the network 
and i = 1,...,M. This procedure produces an overlap 



curve as a function of the number o patterns stored i, 
quantifying the degradation of performance as the num- 
ber of patterns grows. Figure ^ shows the overlap curves 
for several experiments, each averaged over a hundred re- 
alizations, exploring both Barabasi-Albert and Random 
network architectures with varying (fc), implemented by 
changing m. 

To implement hub attack, in each experiment one hub 
is identified and eliminated from W by setting to zero all 
the elements of the connection matrix along the respec- 
tive column and row. We eliminate hubs either sequen- 
tially, i.e., from the more connected to the less connected 
hub, or at random, with uniform probability. Figure [2 
shows four experiments in which 30 hubs, belonging to 
two types of networks (random or Barbasi- Albert), were 
successively attacked in two manners (orderly or not). 
The Barabasi-Albert networks were grown with param- 
eters mo = m = 100. The random network counter- 
parts were obtained with equivalent (k). Both types of 
networks included N = 800 neurons. As expected, the 
overall recall performance decreased with the number of 
trained patterns and as the hubs were eliminated. The 
mean node degree (k) was also directly affected. Marked 
differences appears then in these cases, discriminating the 
impact of hub withdraws, orderly or not, for each archi- 
tecture. Note, for instance, that the Barabasi networks 
are more sensitive to orderly hub removal than a random 
network with equivalent degree. Note also that both net- 
works respond essentially in the same manner to random 
hub deletion. 

The similar effect of random hub removal from both 
network models can be better illustrated by defining a re- 
call index as corresponding to the number of patterns at 
which the respective overlap value reaches 0.8. We also 
define the attack rate as the number of eliminated hubs 
divided by the total number of neurons. Figure [21 shows 
the degradation of the recall index for each experiment 
as a function of the attack rate, showing clearly the dif- 
ferent behavior of a Barabasi network under orderly hub 
elimination. Yet, Figure 0] shows the stability of such 
behavior under variations of network parameters N and 
(k). 

These results support the intuitive expectation about 
hub importance for the inner workings of the Hopfield 
memory model. Note nevertheless that the comparisons 
between Random and Barabasi network were carried out 
regarding only hub attack and, in this manner, it is not 
guaranteed that the same amount of links or edges is 
eliminated from both networks. In what follows, we carry 
out further experiments in order to evaluate the relative 
importance of edges and hub removal. 

We consider three experiments ensuring that the num- 
ber of edges eliminated from each networks is always the 
same. These experiments are illustrated in Figure [S] In 
the first step, the hubs in a Barabasi network are at- 
tacked, as indicated in Figure The black circle rep- 
resents a hub and the dashed line its e eliminated edges. 
From an identical Barabasi network we then eliminate 
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Number of patterns Number of patterns 

FIG. 1: Overlap as a function of the number of patterns for Barabasi- Albert and Random Network with different (k). 
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FIG. 2: Overlap as a function of the number of patterns for Barabasi- Albert and Random networks under both random and 
orderly hub attack . 



e = 6 edges at random, as shown in Figure [5)3. In our ex- 
periments with large networks this procedure is repeated 
until 300 hubs are eliminated. The overlap curve for both 
networks obtained at each step, after another 30 hubs 
were eliminated, is shown in Figure^ A and B). For an 
equivalent Random network, the e edges are eliminated 
either at random or by removing nodes at random until 
the total amount of eliminated edges equals e, as shown 



in Figure 0C and D, respectively. In the actual exper- 
iments, overlaps curves are again produced for each set 
of 30 eliminated hubs. The results for this experiments 
are showed in the Figure |BlC and D, respectively. Fig- 
ure [7| shows the performance of the networks under at- 
tack, quantified by the recall index, in terms of the attack 
ratio. 
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FIG. 3: Recall index as a function of the attack rate for N = 800 and (k) = 160.7 
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FIG. 4: The slope, defined in Figure [3] as a function of two important network parameters 



IV. DISCUSSION AND CONCLUSION 

Although largely overlooked by initial studies in arti- 
ficial neuronal networks, the connectivity between neu- 
rons plays a decisive role in constraining and even defin- 
ing the dynamical properties of the neuronal systems, 
especially the potential of memory recall. The results 
obtained in our investigation substantiate further such a 
phenomenon, with special attention focused on the im- 
portance of hubs to the overall memory recall, as quan- 
tified by the overlap index. The main interesting con- 
clusions identified from our experiments are listed and 
discussed in the following: 

• Effect of connectivity dilution: As expected, all 
types of edge and node dilutions act in order to 
reduce the recall in the Hopfield model, for both 
Barabasi and Random networks. See Figure ^ 

• Ordered hub attack: The removal of hubs, start- 
ing with the most connected ones, from Barabasi 



networks tends to cause more severe loss of recall 
performance than when removing hubs from ran- 
dom networks. This conclusion is implied by the 
fact that the curve for ordered attack to hubs in 
Barabasi networks, see Figure^ correspond to the 
strongest recall deterioration. This reflects the rela- 
tive importance of the hubs in Barabasi- Albert neu- 
ronal networks, as a consequence of the fact that 
hubs in such nets tend to have higher node degree 
(and therefore more edges) than hubs in random 
networks with similar average degree. 

• Random node attack: Random attack to both types 
of networks did not tend to produce different per- 
formance reduction, as indicated by the two en- 
tangled curves representing random attack to both 
types of networks in Figure [7] 

• Localized effect of hub removal: A particulalry in- 
teresting point in the obtained results, for both 
types of network models, regards the fact that the 
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FIG. 5: A - hub attack to Barabasi network (black circle), eliminating e = 13 edges (dashed line), B - edge attack to the same 
Barabasi network eliminating e = 13 edges at random, C - edge attack to Random network eliminating 13 edges at random, D 
- node attack to Random network eliminating nodes until the amount of eliminated edges reaches e = 13, that gives a total of 
5 nodes. 
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random removal of the same quantity of edges as in 
a large hub from the same network has a stronger 
effect on the recall performance than removing that 
hub. This property is supported by the fact that 
the curves, in Figure for hub (node) attack are 
higher than the respective edge attack curves. This 
is a direct consequence of the fact that the removal 
of the hub implies more localized connectivity de- 
terioration, affecting predominantly the single bit 
of the recovered pattern which is directly related 
to the hub. 

All in all, it has become clear that hub and edge attack, 
even when involving the same number of links, have quite 
distinct effects on the memory recall of Hopfield networks 



running on random and Barabasi- Albert networks. 

Future works related to the currently reported investi- 
gation should consider the effect of edge and hub attack 
on other network topologies such as small-world p] and 
Sznajd '9,]. ft would also be interesting to extend such ex- 
perimental analyses to artificial neuronal network models 
such as the perceptron 0] and self-organizing maps [2l| . 
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